System for inspecting objects using augmented reality

ABSTRACT

A method of comparing measured three-dimensional ( 3 D) measurement data to an object is provided. The method includes the steps of providing a three dimensional measurement device configured to measure  3 D coordinates of points on the object and a computing device having a camera and display. During an inspection process, the method measures the object with the  3 D measurement device which provides a first collection of  3 D coordinates. The first collection of  3 D coordinates is stored on the computer network and is associated with an AR marker. During an observation process the method reads the AR marker and transmits from the computer network the first collection of  3 D coordinates and a dimensional representation of the object to the computing device. A portion of the first collection of  3 D coordinates is registered to the camera image. On the integrated display the registered collection of  3 D coordinates and the camera image are shown.

CROSS-REFERENCE TO RELATED APPLICATIONS

The present application a continuation application of U.S. patentapplication Ser. No. 15/081,032 filed on Mar. 25, 2016, which is anonprovisional patent application of U.S. Provisional Application Ser.No. 62/138,433 filed on Mar. 26, 2015, the contents of both of which areincorporated herein by reference.

BACKGROUND

The subject matter disclosed herein relates to the dimensionalinspection of objects using three-dimensional (3D) coordinatemeasurement devices, and more particularly to a system that allows theretrieval of inspection data using augmented reality.

Augmented reality (AR) refers to a direct or indirect view of thephysical, real-world environment whose elements are augmented based oncomputer-generated sensory input (e.g., accelerometer or GPS data). Inthis manner, AR enhances one's current perception of relation. Bycontrast, virtual reality replaces the real world with a simulated one.

AR is generally performed as an application being executed on a portabledevice to display information in context with environmental elements.For example, an AR application running on a portable device may be usedin a golf context to show distances to a fairway or green. Further, anAR application running on a portable device may be used in an outdoorcontext to show compass parameters, position information, or points ofinterest relative to the physical environment in which one resides.Further, an AR application running on a portable device may be used in agaming context, in which objects in the physical environment in whichone resides becomes part of a game experience. With AR,computer-generated graphics (e.g., words and/or lines) are oftenoverlaid onto the real-world view provided by a camera view of aportable device.

While existing systems were suitable for their intended purposes, theneed for improvement remains, particularly in providing a system thatimproves access to inspection data and facilitates operator interactionwith the inspection data using augmented reality.

BRIEF DESCRIPTION

According to one aspect of the invention, a method of comparing measuredthree-dimensional (3D) measurement data to an object is provided. Themethod includes the steps of providing a 3D measurement deviceconfigured to measure 3D coordinates of points on the object; providinga computing device having an integrated camera and an integrateddisplay; providing an AR-marker reader; providing a computer network;and providing a dimensional representation of the object and storing thedimensional representation of the object on the computer network. Duringan inspection process, the method performs steps of: measuring theobject with the 3D measurement device, the 3D measurement deviceproviding a first collection of 3D coordinates; storing the firstcollection of 3D coordinates on the computer network; associating an ARmarker with the object. During an observation process occurring at atime later than the inspection process, the method performs the stepsof: reading the AR marker with the AR-marker reader; transmitting fromthe computer network the first collection of 3D coordinates and thedimensional representation of the object; determining an objectidentification based at least in part on the AR marker; positioning theintegrated camera to view the object and to generate a camera image ofthe viewed object; displaying the camera image on the integrateddisplay; registering a portion of the first collection of 3D coordinatesto the camera image to obtain a registered collection of 3D coordinates,the registration based at least in part on comparison of the cameraimage and one of the first collection of 3D coordinates and thedimensional representation of the object on the computer network; andshowing on the integrated display the registered collection of 3Dcoordinates and the camera image.

These and other advantages and features will become more apparent fromthe following description taken in conjunction with the drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

The subject matter, which is regarded as the invention, is particularlypointed out and distinctly claimed in the claims at the conclusion ofthe specification. The foregoing and other features, and advantages ofthe invention are apparent from the following detailed description takenin conjunction with the accompanying drawings in which:

FIG. 1 is a schematic illustration of an operator performing aninspection in accordance with an embodiment of the invention;

FIG. 2 is an illustration of an operator viewing an inspection report ofan object using augmented reality in accordance with an embodiment ofthe invention;

FIG. 3 is an illustration of an operator viewing an inspection report ofan object using augmented reality in accordance with another embodimentof the invention;

FIG. 4 is an illustration of a portable computing device displaying aninspected object with an inspection report superimposed on the object;

FIG. 5-7 are flow diagrams of methods of acquiring and reviewinginspection reports using augmented reality in accordance withembodiments of the invention; and

FIG. 8 is a flow diagram of a method of guiding an operator in anoperation using augmented reality in accordance with an embodiment ofthe invention.

The detailed description explains embodiments of the invention, togetherwith advantages and features, by way of example with reference to thedrawings.

DETAILED DESCRIPTION

Embodiments of the invention provide advantages in allowing an operatorto review the results of an inspection performed on an object.Embodiments of the invention provide advantages in allowing the operatorto review the inspection results by superimposing the inspection resultson a real-time image of the object. Still further embodiments of theinvention provide advantages in allowing the operator to review theinspection results by superimposing the inspection results on a virtualimage of the object where the virtual image of the object is displayedon a real-time image of the environment.

Embodiments of the invention provide further advantages in storing theinspection results in a network server. The inspection results may bequickly retrieved using a machine readable code or marker that isautomatically generated and associated with the inspected object and themeasured dimensional data. Embodiments of the invention provide stillfurther advantages in allowing the inspection results to be reviewed ona variety of devices.

Referring now to FIG. 1, a system 20 for inspecting an object anddisplaying an inspection report using augmented reality is shown. In theillustrated embodiment, an operator 22 is using a three-dimensional (3D)measurement device 24, such as a portable articulated arm coordinatemeasurement machine (AACMM) for example, to measure the 3D coordinatesof points on a workpiece or object 26. The AACMM may be similar to thatdescribed in commonly owned U.S. Pat. No. 8,533,967 entitled CoordinateMeasurement Machine With Removable Accessories, the contents of which isincorporated herein by reference. It should be appreciated that whileembodiments herein with respect to the illustrated embodiment describethe 3D measurement device 24 as an AACCM, this is for exemplary purposesand the claimed invention should not be so limited. In otherembodiments, the 3D measurement device may be a laser tracker, atriangulation scanner, a structured light scanner, a theodolite, a totalstation, a photogrammetry system, a time-of-flight scanner or acombination of the foregoing.

The 3D measurement device 24 is configured to measure the coordinates ofpoints on the object 26 in response to an action by the operator, suchas pressing an actuator for example. In one embodiment, the 3Dmeasurement device 24 is configured to measure characteristics of theobject 26 in accordance with a predetermined inspection plan. Thecharacteristics may include any typical physical characteristic of theobject 26, such as but not limited to the height, length, and width offeatures of the object, the flatness or curvature of features on theobject or the diameter or depth of holes, slots or recesses for example.The 3D measurement device 24 includes an electronic circuit having aprocessor 50 and memory 48 to allow the 3D measurement device 24 torecord and store in memory 48 the 3D coordinates of the object 26. Inone embodiment, the 3D measurement device 24 may compare the 3Dcoordinates of the object 26 to specifications of the object 26 todetermine if the features are within acceptable tolerances. In oneembodiment, the 3D measurement device 24 may provide an indication tothe operator 22 that a particular feature or dimension of the object 26is outside of the acceptable specification.

In one embodiment, the 3D measurement device 24 is coupled tocommunicate with a local computing device 28, such as a computer forexample. The 3D measurement device and local computing device 28 mayalso be directly connected to a local area network (LAN) or wide areanetwork (WAN) 30. As will be discussed in more detail herein, thenetwork 30 allows for the 3D coordinate data acquired by the 3Dmeasurement device 24 to be stored on a server 40 on the network 30. Theserver 40 stores the data and makes it accessible to a variety ofdevices, such as but not limited to desktop computers 32, cellularphones 34, laptop computers 36, tablet computers or other types ofmobile computing devices. The communication medium that connects 3Dmeasurement device 24 with the local computing device 28 or the network30 may be wired (e.g. Ethernet), wireless (e.g. WiFi or Bluetooth) or acombination of the foregoing. In one embodiment, the local computingdevice 28 may be a tablet computer or a cellular phone. In oneembodiment, coupled to the local computing device 28 is a printer 38.The printer 38 may also be directly coupled to the 3D measurement device24. As will be discussed in more detail herein, the printer 38 is adevice capable of printing a machine readable code or marker, such as aQR code or a bar code for example. Such a machine readable symbol ormarker may also be referred to as an AR marker.

Referring now to FIGS. 2-5 with continuing reference to FIG. 1, a method100 of measuring the object 26 and generating an inspection report thatmay be viewed using an augmented reality display is performed. In method100, the operator 22 first inspects the object 26 in block 102. Theinspection is performed with the 3D measurement device 24 by acquiringthe 3D coordinates of points on the object 26. From the 3D coordinates,the 3D measurement device 24 can determine dimensional characteristicsfor the object 26, such as the length 42 or height of a feature or thediameter 44 of a hole for example. In an embodiment where the 3Dmeasurement device is an AACMM 24, the measurements may be made by theoperator manually moving the arm segments and touching a probe 46 to theobject 26. The inspection steps may be part of a predeterminedinspection plan. In one embodiment, the predetermined inspection plan isstored in memory 48 on the 3D measurement device 24. In anotherembodiment, the inspection plan is a paper document followed by theoperator 22. In one embodiment, the measurements are made at thediscretion of the operator 22 and stored in the 3D measurement device 24memory 48. The 3D measurement device 24 may include an electroniccircuit 50 that determines 3D coordinate data from measurements made bythe 3D measurement device and stores the 3D coordinates in memory 48.For AACMM 24, the 3D coordinate data may be determined from encodersignals located at each of the joints of the arm segments that allow theprocessor 50 to determine the location of the probe 46 in a localcoordinate system when the probe 46 contacts the object 26. In otherembodiments, the 3D coordinate data may be determined by projectinglight (e.g. a laser light, a line of laser light or a structured lightpattern) and the acquisition of images of the projected light on thesurface of the object.

Once the measurements are performed according to the inspection plan,the 3D coordinate data are transmitted to a remote server in block 104,such as server 40 for example. It should be appreciated that the 3Dcoordinate data may be transmitted to the server as it is acquired. Theserver 40 receives the 3D coordinate data and stores the data in memoryor on a data storage device. In one embodiment, the 3D coordinate datamay be stored in a database to facilitate searching and retrieval. Oncethe 3D coordinate data has been received and stored, the server 40executes instructions for generating a machine readable symbol oraugmented reality (AR) marker in block 106. A machine readable symbolmay be a bar code, a QR code or other type of now known or hereinafterdeveloped code, such as but not limited to two-dimensional (2D) Azteccode (ISO/IEC 24778.2008 standard) for example. In one embodiment shownin FIG. 6, the machine readable symbol may be generated in block 112 andtransmitted to a printer device, such as printer 38 for example, that isarranged local to the object 26 in block 114. The printer 38 prints themachine readable symbol 52 on media having an adhesive surface thatallows the printed machine readable symbol 52 to be fixed in block 116to either the object (FIG. 2) or an associated article, such as thedrawings 54 or assembly documents for example. It should be appreciatedthat other machine readable systems may also be used and the term ARmarker also includes passive and active data storage devices such as butnot limited to radio-frequency identification (RFID) and near-fieldcommunication (NFC) circuits or tags for example.

The method 100 then proceeds to block 108 where the 3D coordinates maybe selectively transmitted from the server 40 to another computingdevice, such as by a portable computing device 56. The portablecomputing device 56 may be, but is not limited to a tablet computer, acellular phone, a laptop computer, a personal digital assistant, ane-reader device or the like for example. The computing device includesone or more processors and memory for executing computer readableinstructions. In one embodiment, the 3D coordinates may be recalled by astationary computing device, such as a desktop computer for example,such as through a computer network for example. The computing device 56may include a digital camera 58 having a field of view that isconfigured to acquire a digital image of the area adjacent the computingdevice 56 and display the image on a screen or video display 60. Thecamera 58 may be configured to acquire still images or video images.

In one embodiment, the 3D coordinate data is transmitted from the server40 to the computing device 56 in response to the computing device 56acquiring an image of the machine readable symbol 52. With the 3Dcoordinate data transferred to the computing device 56, the method 100proceeds to block 110 where the 3D coordinate data is superimposed on areal-time or substantially real-time video image to generate anaugmented reality image on the display 60 (FIGS. 3-4). As used herein,an augmented reality image is a combination of a camera image of anarea, such as the work surface 62 for example, with a computer generatedimage. As used herein, the computer generated image may be referred toas a virtual object 64. The virtual object 64 is displayed in a mannerthat makes it appear as if the virtual object 64 is actually present onthe work surface 62. For example other objects, such as a pencil 68, maybe shown on the display and partially or totally obscured by the virtualobject 64. In one embodiment, the virtual object may be located in thecamera image at the location of the machine readable symbol 52 oranother augmented reality marker. The displaying of the virtual object64 may be dynamic, such that as the operator moves the computing device65 around or about the work surface 62, different areas of the virtualobject 64 may be displayed.

The computing device 56 is further configured to display inspectiondata, such as dimensions 42, 44 for example, on the display 60. Thedisplaying of the dimensions may be user selectable or may be automated(e.g. steps through the inspection plan). One or more symbols 66 ortextual information may be displayed alongside the virtual object. Thesesymbols 66 may be used to convey information about the results of theinspection, such as a warning that a feature has a dimension thatdeviates from or is outside of the desired specification or tolerancefor example. In one embodiment, deviations from the specification aredetermined by comparing a collection of 3D coordinate data to adimensional representation of the object, such as acomputer-aided-design (CAD) model for example.

It should be appreciated that the viewing of the virtual object 64 usingaugmented reality may be performed anywhere that the computing device 56may receive the 3D coordinate data. This provides advantages in allowingpersonnel to review inspection results for an object even if theobjected is remotely located.

In another embodiment shown in FIGS. 2 and 4, the computing device 56may superimpose in real-time or substantially real-time the 3Dcoordinate data on the object 26 in response to the machine readablesymbol being acquired by the camera 58 and the object 64 being within(or being brought within) the field of view of the camera 58. Withreference now to FIG. 7 and with continuing reference to FIGS. 1, 2, 4and 5, the process 120 of block 110 (FIG. 5) is shown for an embodimentwhere the 3D coordinate data is displayed on an image of the real oractual object 26. The process 120 includes the step of registering the3D coordinate data to the object 26 in the camera image in block 122. Asused herein, the term “registering” means the transforming the 3Dcoordinate data into a local coordinate system so that the 3D coordinatedata overlays or is superimposed on the same location within the cameraimage as the object 26.

The registration of the 3D coordinate data may be performed in severalways. In one embodiment, the process 120 proceeds to block 124 where the3D coordinate data is registered to the object 26 using the naturalfeatures of the object 26. These natural features may include corners 70(FIG. 4) or edges 72 of features for example. The natural features mayalso include an entire feature, such as the edge of a hole for example.The natural features 70, 72 used in registration may be indicated on thedisplay 60, such as in the form of a symbol or small circle for example.

Instead of using natural features 70, 72, the process 120 may insteadproceed to block 128 where registration is performed using markers 74,such as photogrammetry markers for example, located on the object 26.The use of markers 74 may provide advantages where the 3D measurementdevice is a measurement device such as a triangulation scanner or alaser line probe where images of the object 26, including the markers74, are acquired while the inspection plan is being performed.

The registration process under either the natural feature registrationor the registration marker process results in the generation of aregistered collection of 3D coordinate data. This registered collectionof 3D coordinate data is the 3D coordinate data transformed into thelocal coordinate system of the camera image. After the registrationsteps, the process 120 then proceeds to block 130 where the camera imageis displayed on the display 60. Then at least a portion of the 3Dcoordinate data or inspection data is superimposed or overlaid on thecamera image, such as the dimensions 42, 44 or warning symbol 66.

Embodiments described above refer to a relatively review of theinspection plan, however, the augmented reality display of 3Dcoordinates may also be used to guide the performance of operations onthe object 26. Referring now to FIG. 8, an embodiment of a method 140 isshown for performing an operation on an object. The method 140 starts inblock 142 where an operation for the operator to perform is determined.The operation may be in response to the 3D coordinate data acquired aspart of the inspection plan for example. If the 3D coordinate dataindicates that a feature is outside of the desired specification ortolerance, then the determined operation may allow the feature to bebrought within specification, such as by reaming an undersized hole forexample. It should be appreciated that the determined operation does nothave to be related to a feature that is out of specification. Thedetermined operation may include, but is not limited to: drilling,milling, turning, filing, deburring, welding, soldering, gluing,riveting, screwing, nailing, painting, cutting, and affixing labels forexample.

The method 140 then proceeds to block 144 where the computing deviceprojects a pattern of light onto the object 26 in the location where theoperation is to be performed. The location may be a feature (e.g. ahole) or a position on a surface (e.g. affix a label) for example. Thepattern of light may be generated by the light source used for thecamera 58 for example. In this embodiment, the pattern of light may beprojected only when the computing device 56 is oriented and positionedto allow the pattern to fall on the desired location. The method 140then proceeds to block 146 where the pattern of light or a computergenerated image of the pattern of light is displayed on the display 60.It should be appreciated that the projecting of the pattern of light andthe displaying of the pattern may provide advantages in allowing twodifferent operators to cooperate in performing the operation since theoperator holding the computing device 56 will not have to look away fromthe computing device 56 to see where the pattern is projected. Themethod 140 then proceeds to block 148 where the operation is performedby the operator. The method 140 may optionally loop back to block 144 tosequentially project a pattern of light onto a plurality of locations.In one embodiment, the computing device 56 is sized and shaped to allowthe operator 22 to hold a tool 76 (FIG. 2) for performing the operation.

It should be appreciated that while embodiments herein describe the 3Dcoordinate data as being transmitted to a remote server, this is forexemplary purposes and the claimed invention should not be so limited.In other embodiments, the data may be stored on a computing deviceadjacent or nearby the 3D measurement device. This local computingdevice may then be accessed by other devices via the network. In anotherembodiment, the data is stored on the 3D measurement device and the 3Dmeasurement device functions as a server for the distribution of thedata to remotely located computing devices that use the data to reviewthe inspection report using an augmented reality display.

Technical effects and benefits include the transforming of 3D coordinatedata and inspection plan result data into a visual representation on adisplay that allows a user to view the inspected object as if it waslocated in the area adjacent the user. Further technical effects andbenefits include the guiding of a user in the performance of anoperation by the projection of a light from a portable computing deviceonto the object based at least in part on the 3D coordinate data.

In accordance with one embodiment of the invention a method of comparingmeasured three-dimensional (3D) measurement data to an object isprovided. The method includes the steps of providing a 3D measurementdevice configured to measure 3D coordinates of points on the object;providing a computing device having an integrated camera and anintegrated display; providing an AR-marker reader; providing a computernetwork; and providing a dimensional representation of the object andstoring the dimensional representation of the object on the computernetwork. During an inspection process, the method performs steps of:measuring the object with the 3D measurement device, the 3D measurementdevice providing a first collection of 3D coordinates; storing the firstcollection of 3D coordinates on the computer network; associating an ARmarker with the object. During an observation process occurring at atime later than the inspection process, the method performs the stepsof: reading the AR marker with the AR-marker reader; transmitting fromthe computer network the first collection of 3D coordinates and thedimensional representation of the object; determining an objectidentification based at least in part on the AR marker; positioning theintegrated camera to view the object and to generate a camera image ofthe viewed object; displaying the camera image on the integrateddisplay; registering a portion of the first collection of 3D coordinatesto the camera image to obtain a registered collection of 3D coordinates,the registration based at least in part on comparison of the cameraimage and one of the first collection of 3D coordinates and thedimensional representation of the object on the computer network; andshowing on the integrated display the registered collection of 3Dcoordinates and the camera image.

In accordance with an embodiment, the method may further comprise thesteps of: during the inspection process, determining deviations of thefirst collection of 3D coordinates relative to the dimensionalrepresentation of the object; and during the observation process,showing on the integrated display the deviations of the first collectionof 3D coordinates. The method may further comprise a step wherein thestep of registering a portion of the first collection of 3D coordinatesto the camera image to obtain a registered collection of 3D coordinatesfurther comprises comparing natural features of the object recognized onthe camera image to the dimensional representation of the object.

In accordance with an embodiment, the method may perform a step whereinthe step of registering a portion of the first collection of 3Dcoordinates to the camera image to obtain a registered collection of 3Dcoordinates further comprises comparing a plurality of registrationmarkers on the object as seen in the camera image to a plurality ofregistration markers as observed in the first collection of 3Dcoordinates.

In accordance with an embodiment, the method may perform stepscomprising: providing the computing device with a projector; and duringthe observation process, projecting with the projector a first patternof light. The step of projecting with the projector a first pattern oflight, the projected light is a symbol that indicates a machining orassembly operation to be performed by an operator. The step ofprojecting with the projector a first pattern of light may furtherinclude a symbol that represents the machining or assembly operationthat is selected from the group consisting of drilling, milling,turning, filing, deburring, welding, soldering, gluing, riveting,screwing, nailing, painting, cutting, and affixing labels.

In accordance with an embodiment, the step of projecting with theprojector a first pattern of light, the first pattern of light isdisplayed to draw an observer's attention to a location on the object.The step of projecting with the projector a first pattern of light, thefirst pattern of light is projected sequentially to a plurality oflocations on the object.

In accordance with an embodiment in the step of showing on theintegrated display the registered collection of 3D coordinates and thecamera image, the 3D coordinates are obtained for an object used in themanufacture of goods; and the method may further include a step ofshowing on the integrated display production metric associated with theobject, the production metric selected from a group consisting ofquantity of goods produced, record of maintenance performed, and recordof next calibration due date.

In accordance with an embodiment the 3D measurement device may beselected from the group consisting of a triangulation scanner, atime-of-flight scanner, a laser tracker, an articulated arm coordinatemeasuring machine (CMM), and a Cartesian CMM.

In accordance with an embodiment the AR marker may be selected from thegroup consisting of a barcode marker, a QR-code marker, aradio-frequency-identification (RFID) tag, and a near-fieldcommunication (NFC) tag.

The terminology used herein is for the purpose of describing particularembodiments only and is not intended to be limiting of the invention. Asused herein, the singular forms “a”, “an” and “the” are intended toinclude the plural forms as well, unless the context clearly indicatesotherwise. It will be further understood that the terms “comprises”and/or “comprising,” when used in this specification, specify thepresence of stated features, integers, steps, operations, elements,and/or components, but do not preclude the presence or addition of oneor more other features, integers, steps, operations, element components,and/or groups thereof.

The corresponding structures, materials, acts, and equivalents of allmeans or step plus function elements in the claims below are intended toinclude any structure, material, or act for performing the function incombination with other claimed elements as specifically claimed. Thedescription of the present invention has been presented for purposes ofillustration and description, but is not intended to be exhaustive orlimited to the invention in the form disclosed. Many modifications andvariations will be apparent to those of ordinary skill in the artwithout departing from the scope and spirit of the invention. Theembodiment was chosen and described in order to best explain theprinciples of the invention and the practical application, and to enableothers of ordinary skill in the art to understand the invention forvarious embodiments with various modifications as are suited to theparticular use contemplated.

The present invention may be a system, a method, and/or a computerprogram product. The computer program product may include a computerreadable storage medium (or media) having computer readable programinstructions thereon for causing a processor to carry out aspects of thepresent invention.

The computer readable storage medium can be a tangible device that canretain and store instructions for use by an instruction executiondevice. The computer readable storage medium may be, for example, but isnot limited to, an electronic storage device, a magnetic storage device,an optical storage device, an electromagnetic storage device, asemiconductor storage device, or any suitable combination of theforegoing. A non-exhaustive list of more specific examples of thecomputer readable storage medium includes the following: a portablecomputer diskette, a hard disk, a random access memory (RAM), aread-only memory (ROM), an erasable programmable read-only memory (EPROMor Flash memory), a static random access memory (SRAM), a portablecompact disc read-only memory (CD-ROM), a digital versatile disk (DVD),a memory stick, a floppy disk, a mechanically encoded device such aspunch-cards or raised structures in a groove having instructionsrecorded thereon, and any suitable combination of the foregoing. Acomputer readable storage medium, as used herein, is not to be construedas being transitory signals per se, such as radio waves or other freelypropagating electromagnetic waves, electromagnetic waves propagatingthrough a waveguide or other transmission media (e.g., light pulsespassing through a fiber-optic cable), or electrical signals transmittedthrough a wire.

Computer readable program instructions described herein can bedownloaded to respective computing/processing devices from a computerreadable storage medium or to an external computer or external storagedevice via a network, for example, the Internet, a local area network, awide area network and/or a wireless network. The network may comprisecopper transmission cables, optical transmission fibers, wirelesstransmission, routers, firewalls, switches, gateway computers and/oredge servers. A network adapter card or network interface in eachcomputing/processing device receives computer readable programinstructions from the network and forwards the computer readable programinstructions for storage in a computer readable storage medium withinthe respective computing/processing device.

Computer readable program instructions for carrying out operations ofthe present invention may be assembler instructions,instruction-set-architecture (ISA) instructions, machine instructions,machine dependent instructions, microcode, firmware instructions,state-setting data, or either source code or object code written in anycombination of one or more programming languages, including an objectoriented programming language such as Smalltalk, C++ or the like, andconventional procedural programming languages, such as the “C”programming language or similar programming languages. The computerreadable program instructions may execute entirely on the user'scomputer, partly on the user's computer, as a stand-alone softwarepackage, partly on the user's computer and partly on a remote computeror entirely on the remote computer or server. In the latter scenario,the remote computer may be connected to the user's computer through anytype of network, including a local area network (LAN) or a wide areanetwork (WAN), or the connection may be made to an external computer(for example, through the Internet using an Internet Service Provider).In some embodiments, electronic circuitry including, for example,programmable logic circuitry, field-programmable gate arrays (FPGA), orprogrammable logic arrays (PLA) may execute the computer readableprogram instructions by utilizing state information of the computerreadable program instructions to personalize the electronic circuitry,in order to perform aspects of the present invention

Aspects of the present invention are described herein with reference toflowchart illustrations and/or block diagrams of methods, apparatus(systems), and computer program products according to embodiments of theinvention. It will be understood that each block of the flowchartillustrations and/or block diagrams, and combinations of blocks in theflowchart illustrations and/or block diagrams, can be implemented bycomputer readable program instructions.

These computer readable program instructions may be provided to aprocessor of a general purpose computer, special purpose computer, orother programmable data processing apparatus to produce a machine, suchthat the instructions, which execute via the processor of the computeror other programmable data processing apparatus, create means forimplementing the functions/acts specified in the flowchart and/or blockdiagram block or blocks. These computer readable program instructionsmay also be stored in a computer readable storage medium that can directa computer, a programmable data processing apparatus, and/or otherdevices to function in a particular manner, such that the computerreadable storage medium having instructions stored therein comprises anarticle of manufacture including instructions which implement aspects ofthe function/act specified in the flowchart and/or block diagram blockor blocks.

The computer readable program instructions may also be loaded onto acomputer, other programmable data processing apparatus, or other deviceto cause a series of operational steps to be performed on the computer,other programmable apparatus or other device to produce a computerimplemented process, such that the instructions which execute on thecomputer, other programmable apparatus, or other device implement thefunctions/acts specified in the flowchart and/or block diagram block orblocks.

The flowchart and block diagrams in the Figures illustrate thearchitecture, functionality, and operation of possible implementationsof systems, methods, and computer program products according to variousembodiments of the present invention. In this regard, each block in theflowchart or block diagrams may represent a module, segment, or portionof instructions, which comprises one or more executable instructions forimplementing the specified logical function(s). In some alternativeimplementations, the functions noted in the block may occur out of theorder noted in the figures. For example, two blocks shown in successionmay, in fact, be executed substantially concurrently, or the blocks maysometimes be executed in the reverse order, depending upon thefunctionality involved. It will also be noted that each block of theblock diagrams and/or flowchart illustration, and combinations of blocksin the block diagrams and/or flowchart illustration, can be implementedby special purpose hardware-based systems that perform the specifiedfunctions or acts or carry out combinations of special purpose hardwareand computer instructions.

While the invention has been described in detail in connection with onlya limited number of embodiments, it should be readily understood thatthe invention is not limited to such disclosed embodiments. Rather, theinvention can be modified to incorporate any number of variations,alterations, substitutions or equivalent arrangements not heretoforedescribed, but which are commensurate with the spirit and scope of theinvention. Additionally, while various embodiments of the invention havebeen described, it is to be understood that aspects of the invention mayinclude only some of the described embodiments. Accordingly, theinvention is not to be seen as limited by the foregoing description, butis only limited by the scope of the appended claims.

1. A method of comparing measured three-dimensional (3D) measurementdata to an object comprising: measuring a first collection of 3Dcoordinates on the object with a 3D measurement device; associating anAR marker with the object; reading the AR marker with an AR-markerreader and, in response, transmitting the first collection of 3Dcoordinates and an electronic dimensional representation of the objectto a computing device having a camera and a display; positioning thecamera to view the object and to generate a camera image of the object;displaying the camera image on the integrated display; registering aportion of the first collection of 3D coordinates to the camera image toobtain a registered collection of 3D coordinates, the registration basedat least in part on comparison of the camera image and one of the firstcollection of 3D coordinates and the dimensional representation of theobject; and showing on the display the registered collection of 3Dcoordinates and the camera image.
 2. The method of claim 1 furthercomprising: determining deviations of the first collection of 3Dcoordinates relative to the dimensional representation of the object;and showing on the integrated display the deviations of the firstcollection of 3D coordinates.
 3. The method of claim 1 wherein theregistering the portion of the first collection of 3D coordinates to thecamera image further comprises comparing natural features of the objectrecognized on the camera image to the dimensional representation of theobject.
 4. The method of claim 1 wherein the registering the portion ofthe first collection of 3D coordinates to the camera image furthercomprises comparing a plurality of registration markers on the object asseen in the camera image to a plurality of registration markers asobserved in the first collection of 3D coordinates.
 5. The method ofclaim 1 further comprising projecting with a projector operably coupledto the computing device a first pattern of light.
 6. The method of claim5 wherein, in the projecting with the projector the first pattern oflight, the first pattern of light includes a symbol that indicates amachining or assembly operation to be performed by an operator.
 7. Themethod of claim 6 wherein, in the symbol indicates the machining orassembly operation selected from the group consisting of drilling,milling, turning, filing, deburring, welding, soldering, gluing,riveting, screwing, nailing, painting, cutting, and affixing labels. 8.The method of claim 5 wherein, in projecting with the projector thefirst pattern of light, the first pattern of light is projected onto apredetermined location on the object.
 9. The method of claim 8 wherein,in the projecting with the projector a first pattern of light, the firstpattern of light is projected sequentially to a plurality ofpredetermined locations on the object.
 10. The method of claim 1 furthercomprising showing on the integrated display a production metricassociated with the object, the production metric selected from a groupconsisting of quantity of goods produced, record of maintenanceperformed, and record of next calibration due date.
 11. The method ofclaim 1, wherein the 3D measurement device is selected from the groupconsisting of a triangulation scanner, a time-of-flight scanner, a lasertracker, an articulated arm coordinate measuring machine (CMM), and aCartesian CMM.
 12. The method of claim 1, wherein the AR marker isselected from the group consisting of a barcode marker, a QR-codemarker, a radio-frequency-identification (RFID) tag, and a near-fieldcommunication (NFC) tag.
 13. A system comprising: an AR-marker readeroperably coupled to the 3D measurement device; a computing device havinga camera and a display, the computing device being operably coupled tothe AR-marker reader the computing device being operably coupled to amemory having computer readable instructions and one or more processorsfor executing the computer readable instructions, the computer readableinstructions comprising: reading an AR marker with the AR-marker readerand, in response, retrieve from the memory a first collection of 3Dcoordinates and an electronic dimensional representation of the object,the first collection of 3D coordinates being measured by a 3Dmeasurement device; acquiring a camera image of the object with thecamera; displaying the camera image on the display; registering aportion of the first collection of 3D coordinates to the camera image toobtain a registered collection of 3D coordinates, the registration basedat least in part on comparison of the camera image and one of the firstcollection of 3D coordinates and the electronic dimensionalrepresentation of the object; and showing on the display the registeredcollection of 3D.
 14. The system of claim 13, wherein the computerreadable instructions further comprise: determining deviations of thefirst collection of 3D coordinates relative to the electronicdimensional representation of the object; and showing on the displaydeviations of the first collection of 3D coordinates.
 15. The system ofclaim 13, wherein the registering the portion of the first collection of3D coordinates to the camera image further comprises comparing naturalfeatures of the object identified on the camera image to the electronicdimensional representation of the object.
 16. The system of claim 13,wherein the step of registering the portion of the first collection of3D coordinates to the camera image further comprises comparing aplurality of registration markers on the object in the camera image to aplurality of registration markers in the first collection of 3Dcoordinates.
 17. The system of claim 13, wherein the computer readableinstructions further comprise projecting with a projector a firstpattern of light.
 18. The system of claim 17 wherein, in the projectingwith the projector the first pattern of light, the first pattern oflight includes a symbol to indicate a machining or assembly operationperformable by an operator.
 19. The system of claim 18 wherein, inprojecting of the symbol, the symbol indicates the machining or assemblyoperation selected from the group consisting of drilling, milling,turning, filing, deburring, welding, soldering, gluing, riveting,screwing, nailing, painting, cutting, and affixing labels.
 20. Thesystem of claim 13 wherein the computer readable instructions furthercomprise a showing on the display a production metric associated withthe object, the production metric selected from a group consisting ofquantity of goods produced, record of maintenance performed, and recordof next calibration due date.